Image processing apparatus, image capturing apparatus, control method and recording medium

ABSTRACT

In an image processing apparatus of the present invention, multiple subjects are selected in an image signal, and subject distances for focusing on each of the selected subjects are specified. A subject distance range defined by the specified subject distances is then set. Then, based on the image signal, the image processing apparatus generates a reconstructed image that is focused on each of the subjects that are to be focused on in the set subject distance range.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of application Ser. No. 14/168,441,filed Jan. 30, 2014, the entire disclosure of which is herebyincorporated by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image processing apparatus, an imagecapturing apparatus, a control method, and a recording medium, andparticularly to a technique for generating an image that is focused atan arbitrary subject distance based on output data after shooting.

Description of the Related Art

In recent years, a technique has been proposed in which, in imagecapturing apparatuses such as digital cameras, an intensity distributionand traveling directions of light is recorded as output data duringshooting so as to make it possible to generate an image, in which afocus position is positioned on an arbitrary image plane, based on theoutput data after recording.

Ren. Ng et al., “Light Field Photography with a Hand-held PlenopticCamera”, Stanford University Computer Science Tech Report CTSR, 2005-02discloses a method (Light Field Photography) in which light that isincident from various directions is recorded separately by causing lightbeams that pass through different divided pupil regions of an imaginglens to form images on the pixels (photoelectric conversion elements) ofan image sensor via a microlens array. In the output data obtained inthis way (light field data, which is hereinafter referred to as “LFdata”), adjacent pixels record light beams that were incident fromdifferent directions.

By extracting light beams from the same direction from the pixelsassociated with each microlens, it is possible to generate an imagecaptured from that direction based on the LF data. Also, by setting afocus position and adding up the output of the pixels that recorded thelight beams that passed through one point in the focal plane whichincludes the focus position, it is possible to virtually generate(reconstruct) pixels for an image that is focused at a specified focusposition (an image plane) after shooting.

Also, by using only pixels that received light beams that passed througha limited range of divided pupil regions in the LF data, it is possibleto obtain an effect equivalent to the case where an aperture is appliedin the imaging optical system. Specifically, by generating areconstructed image using only pixels that correspond to divided pupilregions that correspond to the center of the exit pupil, it is alsopossible to generate an image that has a deep depth of field and isfocused at a wide range of subject distances.

However, when a reconstructed image having a deep depth of field isgenerated as described above, there are cases where a subject other thanthe desired subject is also focused on. For example, a passerby or thelike who is passing in front of or behind the person who is the desiredsubject can possibly be in-focus. In particular, a subject that islocated closer to the image capturing apparatus during shooting than thedesired subject is will have a greater presence in the reconstructedimage.

SUMMARY OF THE INVENTION

The present invention was made in view of such problems in theconventional technique. The present invention provides an imageprocessing apparatus, an image capturing apparatus, a control method,and a recording medium for generating an image that is favorably focusedon multiple desired subjects.

The present invention in its first aspect provides an image processingapparatus comprising: an obtaining unit which is able to obtain datafrom which reconstructed images which are focused on arbitrary imageplanes are able to generate; a display unit which is able to displayimages to a display medium; a selection unit which is able to select aplurality of subjects on a image displayed by the display unit; asetting unit which is able to specify focus positions corresponding tothe plurality of subjects selected by the selection unit and set a focusrange defined by the specified focus positions; and a generation unitwhich is able to generate, based on the data, a reconstructed image inwhich each subject corresponding to the focus range set by the settingunit is focused.

The present invention in its second aspect provides an image capturingapparatus comprising: an imaging unit which is able to output data fromwhich reconstructed images which are focused on arbitrary image planesare able to generate; a display unit which is able to display images toa display medium; a selection unit which is able to select a pluralityof subjects on a image displayed by the display unit; a setting unitwhich is able to specify focus positions corresponding to the pluralityof subjects selected by the selection unit and set a focus range definedby the specified focus positions; and a generation unit which is able togenerate, based on the data, a reconstructed image in which each subjectcorresponding to the focus range set by the setting unit is focused.

The present invention in its third aspect provides a control method ofan image processing apparatus, comprising: an obtaining step in which anobtaining unit of the image processing apparatus obtains data from whichreconstructed images which are focused on arbitrary image planes areable to generate; a display step in which a display unit of the imageprocessing apparatus displays images to a display medium; a selectionstep in which a selection unit of the image processing apparatus selectsa plurality of subjects on a image displayed in the display step; asetting step in which a setting unit of the image processing apparatusspecifies focus positions corresponding to the plurality of subjectsselected in the selection step and set a focus range defined by thespecified focus positions; and a generation step in which a generationunit of the image processing apparatus generates, based on the data, areconstructed image in which each subject corresponding to the focusrange set in the setting step is focused.

The present invention in its fourth aspect provides a control method ofan image capturing apparatus, comprising: an imaging step in which animaging unit of the image capturing apparatus outputs data from whichreconstructed images which are focused on arbitrary image planes areable to generate; a display step in which a display unit of the imagecapturing apparatus displays images to a display medium; a selectionstep in which a selection unit of the image capturing apparatus selectsa plurality of subjects on a image displayed in the display step; asetting step in which a setting unit of the image capturing apparatusspecifies focus positions corresponding to the plurality of subjectsselected in the selection step and set a focus range defined by thespecified focus positions; and a generation step in which a generationunit of the image capturing apparatus generates, based on the data, areconstructed image in which each subject corresponding to the focusrange set in the setting step is focused.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a functional configuration of adigital camera 100 according to an embodiment of the present invention.

FIG. 2 is a diagram for describing the relationship between a microlensarray 105 and an imaging unit 106 according to the embodiment of thepresent invention.

FIG. 3 is a diagram for describing the relationship between light beamsthat pass through various areas of an exit pupil 301 and photoelectricconversion elements for subjecting the light beams to photoelectricconversion according to the embodiment of the present invention.

FIGS. 4A and 4B are diagrams showing the correspondence between areas ofthe exit pupil 301 and photoelectric conversion elements that correspondto microlenses according to the embodiment of the present invention.

FIG. 5 is a diagram for describing the relationship that light beamsthat pass through a specified position in a reconstruction plane havewith pass-through positions in the imaging plane according to theembodiment of the present invention.

FIG. 6 is a flowchart illustrating reconstructed image generationprocessing that is executed in the digital camera 100 according to theembodiment of the present invention.

FIG. 7 is a flowchart illustrating subject selection processing that isexecuted in the digital camera 100 according to the embodiment of thepresent invention.

FIGS. 8A and 8B are diagrams for describing target LF data according tothe embodiment of the present invention.

FIG. 9 is a histogram showing a distribution for a D area according tothe embodiment of the present invention.

FIGS. 10A, 10B, 10C, and 10D are diagrams for describing the generationof an output image according to the embodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, an illustrative embodiment of the present invention will bedescribed in detail with reference to the drawings. Note that thefollowing embodiment is described taking the example in which thepresent invention is applied to a digital camera that can generate animage that is focused on a subject positioned at an arbitrary subjectdistance based on LF data after shooting, as one example of an imageprocessing apparatus. However, the present invention is applicable toany device that can generate an image that is focused on a subjectpositioned at an arbitrary subject distance based on LF data.

Also, the following terms used in the present specification are definedas follows.

“Light Field (LF) Data”

LF data refers to an image signal that is output from an imaging unit106 of the digital camera 100 of the present embodiment. The pixels ofthe image signal indicate signal intensities that correspond to lightbeams having different combinations of an incident direction and a pupilregion of an imaging optical system 104 through which the light beampassed. LF data is also sometimes called “ray space information”.

“Reconstructed Image”

A reconstructed image is an image that is generated from LF data and isfocused on a subject positioned at an arbitrary subject distance.Specifically, the pixel values of the pixels in the focal plane thatcorresponds to the subject distance at which the image is to begenerated (i.e., the pixels in a reconstruction plane) are obtained bycombining the pixel values of the pixels in the LF data that correspondto the light beams that passed through those pixels in accordance withthe pixel arrangement in the focal plane. The pixel arrangement in thereconstruction plane is determined based on the incident directions ofthe light beams that would be incident if the image sensor were locatedin the reconstruction plane. One pixel in the reconstructed image can begenerated by obtaining the sum of the pixel values of the pixels thatcorrespond to one microlens in the pixel arrangement.

Configuration of Digital Camera 100

FIG. 1 is a block diagram showing the functional configuration of thedigital camera 100 according to the present embodiment of the presentinvention.

A control unit 101 is a CPU, for example, and controls operations ofvarious blocks of the digital camera 100. Specifically, the control unit101 controls operations of the blocks by reading out an operationprogram for later-described shooting processing or refocused movingimage generation processing that is stored in a ROM 102, deploying it toa RAM 103, and executing it.

The ROM 102 is a rewritable nonvolatile memory, for example, and storesoperation programs for blocks of the digital camera 100, as well asparameters necessary for operations of the blocks, and the like.

The RAM 103 is a volatile memory. The RAM 103 is used not only as anarea for deployment of operation programs for blocks of the digitalcamera 100, but also as a storage area for storing intermediate dataoutput in operations of the blocks, and the like.

The imaging unit 106 is an image sensor such as a CCD or a CMOS sensor.The imaging unit 106 receives a timing signal that is output from atiming generator (TG) (not shown) in accordance with an instruction fromthe control unit 101, performs photoelectric conversion on an opticalimage that was formed on the photoelectric conversion element face ofthe image sensor by an imaging optical system 104, and outputs an analogimage signal. Note that the imaging optical system 104 includes anobjective lens, a focusing lens, a diaphragm, and the like. Also, thedigital camera 100 of the present embodiment has microlenses that areprovided on the photoelectric conversion elements of the image sensor,and also separately has a microlens array 105 between the image sensorand the imaging optical system 104 on the optical axis.

Relationship Between Microlenses and Photoelectric Conversion Elements

The following describes the microlens array 105 that is provided betweenthe image sensor and the imaging optical system 104 on the optical axisin the digital camera 100 of the present embodiment, with reference tothe drawings.

As shown in FIG. 2, the microlens array 105 of the present embodiment isconstituted by multiple microlenses 201. In FIG. 2, the z axis is theoptical axis of the imaging optical system 104, the x axis is thehorizontal direction when the digital camera 100 is in the landscapeorientation, and the y axis is the vertical direction. Note that for thesake of simplicity in the example shown in FIG. 2, the microlens array105 will be described as being constituted by microlenses 201 arrangedin five rows and five columns, but the configuration of the microlensarray 105 is not limited to this.

Also, FIG. 2 shows an example in which photoelectric conversion elements202 of the image sensor constituting the imaging unit 106 are indicatedby a grid. Each microlens 201 corresponds to a predetermined number ofphotoelectric conversion elements 202, and in the example shown in FIG.2, one microlens 201 corresponds to the photoelectric conversionelements 202 of 5×5=25 pixels. Light beams that pass through onemicrolens 201 are separated according to their incident direction, andform an image on the corresponding photoelectric conversion elements202.

FIG. 3 shows light beams that are incident on photoelectric conversionelements 202 p₁ to p₅ that correspond to one microlens 201. In FIG. 3,the upward direction corresponds to the vertically upward direction.This figure illustrates the light paths of light beams that are incidenton the photoelectric conversion elements 202 as viewed laterally whenthe digital camera 100 is in the landscape orientation. As shown in thisfigure, the light beams that are incident on the horizontally-alignedphotoelectric conversion elements 202 p₁ to p₅ are light beams thatpassed through five areas a₁ to a₅ obtained by dividing an exit pupil301 of the imaging optical system 104 vertically, and then passedthrough one microlens 201. Note that the numbers appended to the areasindicate the correspondence relationship with the photoelectricconversion elements 202 that receive the light beams that passed throughthose areas.

Note that although FIG. 3 shows an example of light paths of light beamsthat are incident on the photoelectric conversion elements 202 as viewedlaterally, the separation of light beams is not limited to the verticaldirection, and separation is similarly performed in the horizontaldirection as well. Specifically, in the case where the exit pupil 301 ofthe imaging optical system 104 is divided into the areas shown in FIG.4A as viewed from the image sensor side, light beams that pass throughthe various areas are incident on the photoelectric conversion elementsthat are appended with the same identification number among thephotoelectric conversion elements 202 shown in FIG. 4B. Note that it isassumed here that the imaging optical system 104 and the microlenses inthe microlens array 105 have substantially the same F-number (F-value).

An AFE 107 and a DFE 108 perform correction processing and the like onan image signal generated by the imaging unit 106. Specifically, the AFE107 performs reference level adjustment (clamp processing) and A/Dconversion processing on an analog image signal that is output from theimaging unit 106, and outputs LF data to the DFE 108. The DFE 108corrects minor reference level deviation and the like in the input LFdata.

An image processing unit 109 applies various types of image processingsuch as color conversion processing to LF data that was subjected tocorrection processing by the DFE 108. In the present embodiment, theimage processing unit 109 also performs processing for generating animage that is focused on a subject positioned at an arbitrary subjectdistance (i.e., a reconstructed image) based on LF data. Thereconstructed image can be generated using a “Light Field Photography”technique such as the previously-described technique disclosed in Ren.

Reconstructed Image Generation Method

The following describes an overview of a method of generating areconstructed image that is focused on a subject at a specified subjectdistance, with reference to the drawings.

First, the subject distance for focusing on a specified subject includedin the shooting range can be acquired with the following method.Firstly, the image processing unit 109 generates images respectivelycorresponding to two light beams that passed through different dividedpupil regions from the LF data, and detects the difference (defocusamount) between the images of the specified subject in the images. Thecontrol unit 101 can then calculate the subject distance for focusing onthe specified subject based on the defocus amount detected in this way.

For each microlens in the example shown in FIG. 4B, it is possible togenerate an image A that corresponds to divided pupil regions in theleft half of the exit pupil 301 by adding up the pixel values of thepixels in the first and second columns among the corresponding pixels.Also, it is possible to generate an image B that corresponds to dividedpupil regions in the right half of the exit pupil 301 by adding up thepixel values of the pixels in the fourth and fifth columns among thecorresponding pixels. This can be represented by the followingexpression.

$\{ {\begin{matrix}{\sum\limits_{a = 1}^{5}{\sum\limits_{b = 1}^{2}( P_{ab} )}} & {{left}\mspace{14mu}{half}\mspace{14mu}{regions}} \\{\sum\limits_{a = 1}^{5}{\sum\limits_{b = 4}^{5}( P_{ab} )}} & {{right}\mspace{14mu}{half}\mspace{14mu}{regions}}\end{matrix}\quad} $

The two types of reconstructed images obtained in this way are imagesfor which the optical axis is the barycentric position of thecorresponding divided pupil regions.

Specifically, since the two types of reconstructed images have imagedeviation due to optical axis deviation, it is possible to detect theimage deviation amount (pupil division phase difference) for eachsubject by performing a correlation calculation with respect to the twoimages. The subject distances for focusing on the subjects included inthe shooting range of the LF data can be analyzed based on imagedeviation amounts obtained in this way, and it is possible to generate areconstructed image that is focused on a specified subject as anadditional image, for example.

The following describes the generation of a reconstructed image that isfocused on a subject at a specified subject distance. In the digitalcamera 100 of the present embodiment, each of the pixels that correspondto one microlens as described above receives a light beam that passedthrough a different divided pupil region of the exit pupil of theimaging lens. The same follows for all of the microlenses in themicrolens array 105. Also, since each microlens receives light beamsthat passed through the imaging lens from a different direction, everypixel of the image sensor receives light beams that were incident fromall of the different directions.

For this reason, the light beams that are incident on the pixels in theLF data obtained by shooting are defined in the following descriptionusing a light path indicated by the coordinates (u,v) of the pupilregion that was passed through in the exit pupil and the positioncoordinates (x′,y′) of the corresponding microlens. In the generation ofa reconstructed image, it is possible to obtain the pixel value of apixel (x,y) in the reconstruction plane that corresponds to thearbitrary subject distance at which the reconstructed image is to begenerated, by integrating the light beams that have light paths thatpass through that point.

FIG. 5 shows light paths of light beams in the horizontal plane (xzplane) as viewed from the vertical direction when the digital camera 100is in the landscape orientation. Although the following describes lightpaths of light beams that pass through pixels in the reconstructionplane in the xz plane, the same follows for the yz plane as well.

Letting (u,v) represent the coordinates of a pupil region, and (x,y)represent the coordinates of a pixel in the reconstruction plane, thefollowing expression obtains the position coordinates (x′,y′) of themicrolens in the microlens array 105 that receives the light beam thatpasses through that divided pupil region and that pixel in thereconstruction plane.

$( {x^{\prime},y^{\prime}} ) = ( {{u + \frac{x - u}{\alpha}},{v + \frac{y - v}{\alpha}}} )$

Note that F represents the distance from the imaging lens to themicrolens array, and αF represents the distance from the shooting lensto the reconstruction plane (α being a refocusing coefficient, which isa variable coefficient for determining the distance to thereconstruction plane).

Also, letting L(x′,y′,u,v) represent the output of the photoelectricconversion element that receives this light beam, the output E(x,y) ofthe pixel at the coordinates (x,y) in the image to be formed in thereconstruction plane is obtained by integrating L(x′,y′,u,v) withrespect to the pupil regions of the shooting lens, as shown by thefollowing expression.

${E( {x,y} )} = {\frac{1}{\alpha^{2}F^{2}}{\int{\int{{L( {{u + \frac{x - u}{\alpha}},{v + \frac{y - v}{\alpha}},u,v} )}{dudv}}}}}$

Note that this expression can be solved through simple addition ifrepresentative coordinates of the pupil region are set as (u,v).

A display unit 110 is a display device such as a compact LCD that thedigital camera 100 has. The display unit 110 displays an image that wasgenerated by the image processing unit 109 and is focused at anarbitrary subject distance. As described above, in the LF data obtainedby performing A/D conversion on an analog image signal output from theimaging unit 106 of the present embodiment, images at adjacent pixelsare not linked. For this reason, image data generated by the imageprocessing unit 109 is displayed on the display unit 110 rather than theLF data.

A recording medium 111 is, for example, an internal memory of thedigital camera 100, or a removable recording device such as a memorycard or an HDD that is connected to the digital camera 100. Therecording medium 111 records LF data as well as images that have beengenerated from the LF data and are focused at arbitrary image planes.

An operation input unit 112 is a user interface of the digital camera100 such as a power button, a shutter button, or the like. Upondetecting that the user interface was operated by the user, theoperation input unit 112 outputs a control signal that corresponds todetected operation to the control unit 101.

Reconstructed Image Generation Processing

The following is a specific description of processing in thereconstructed image generation processing executed in the digital camera100 of the present embodiment having the above-described configuration,with reference to the flowchart of FIG. 6. Processing corresponding tothis flowchart can be realized by the control unit 101 reading out acorresponding processing program stored in the ROM 102, deploying it tothe RAM 103, and executing it, for example. Note that the reconstructedimage generation processing will be described as processing that isstarted when the digital camera 100 is set to the browsing mode andstarted up, and then the user gives an instruction to generate areconstructed image from LF data recorded on the recording medium 111,for example.

In step S601, the control unit 101 sets distance information acquisitionareas (referred to hereinafter as “D areas”) for acquiring correspondingpieces of information indicating the distances to subjects from thetarget LF data for which the generation instruction was given. A D areais a two-dimensional area in the pixel array constituting the target LFdata, and is defined in units of the group of pixels (photoelectricconversion elements 202) that corresponds to one microlens 201.Specifically, the size of the D area is defined according to the minimumresolution of the reconstructed image that can be generated from the LFdata. For example, in the example shown in FIG. 2, the group of 5×5pixels that corresponds to one microlens 201 corresponds to the minimumreconstructed image resolution, and therefore the D area is defined asan area whose number of pixels in the horizontal direction is a multipleof 5, and whose number of pixels in the vertical direction is a multipleof 5.

Note that a configuration is possible in which the D area is defined inunits of the group of pixels that corresponds to the minimum resolutionof the reconstructed image that can be generated from the target LFdata, and the size of the D area is set to an appropriate size accordingto the precision required for the distance to a subject, and limitationssuch as the device computing power, the calculation amount, and therequired frame rate.

In step S602, for each of the D areas that were set in step S601, thecontrol unit 101 calculates information indicating a representativedistance to subjects included in the area. Specifically, for each of theD areas of the target LF data, the control unit 101 causes the imageprocessing unit 109 to generate two types of reconstructed images fordefocus amount detection (detection images) that correspond to lightbeams that passed through two different divided pupil regions. Thedetection images that are generated in this step may correspond to lightbeams that passed through divided pupil regions obtained by dividing theregion of the exit pupil 301 into left-half and right-half divided pupilregions as previously described. However, the implementation of thepresent invention is not limited to this; it is only necessary that thedetection images are two types of images that correspond to, from amongthe light beams that passed through the exit pupil 301, light beams thatpassed through two types of divided pupil regions that have differentoptical axes, and the method of selecting divided pupil regions is notlimited to this.

The control unit 101 then calculates a representative distance for eachD area in accordance with the result of analyzing the obtained defocusamounts. Note that the representative distance may be the distancerelated to the subject located in the center of the corresponding Darea, or may be an average value of distances obtained for all of thesubjects in the corresponding area, for example.

In step S603, the control unit 101 executes subject selection processingfor allowing the user to select subjects (in-focus subjects) that are tobe focused on in the reconstructed image that is ultimately generated(i.e., the final image).

Subject Selection Processing

The following describes details of the subject selection processingexecuted in step S603 with reference to the flowchart of FIG. 7.

In step S701, under control of the control unit 101, the imageprocessing unit 109 generates a selection reconstructed image forallowing the user to designate in-focus subjects that are to be focusedon in the final image. The selection reconstructed image may be an imagegenerated so as to be focused at the subject distance (an image plane)that was set when the target LF data was obtained, for example.

Alternatively, in order for the subjects to be recognized easily, theselection reconstructed image may be an image with a deep depth of fieldthat is generated using pixels that correspond to the light beams thatpassed through a limited range of divided pupil regions, such as thecentral portion of the exit pupil 301.

In step S702, the control unit 101 transmits the selection reconstructedimage that was generated by the image processing unit 109 to the displayunit 110, and causes it to be displayed along with a message instructingthe user to select in-focus subjects. The display unit 110 of thepresent embodiment may be a touch panel display that includes a sensorfor detecting touch operations, for example, and the user may be allowedto select in-focus subjects by performing touch operations on theselection reconstructed image displayed on the display unit 110. When atouch operation is performed, a touch operation detection signal istransmitted to the operation input unit 112, and the operation inputunit 112 converts the detection signal into a control signal and outputsthe control signal to the control unit 101.

In step S703, the control unit 101 determines whether or not a firstin-focus subject selection was made. In the present embodiment, it isassumed that the digital camera 100 has a function of detecting areasthat correspond to people's faces included in images, and it is assumedthat in step S703, it is determined whether or not the user selected aface area that is to be the in-focus subject. Note that the in-focussubject is not limited to being a person's face area, and it may be aspecified subject that the digital camera 100 has a detection functionfor. Also, although the in-focus subject is described as being aperson's face area in the present embodiment, the implementation of thepresent invention is not limited to this, and a subject such as abuilding located at a position designated by the user may be set as thein-focus subject.

If the control unit 101 determines that a first in-focus subjectselection was made, the procedure moves to step S704, and if itdetermines that a first in-focus subject selection has not been made,the processing of step S703 is repeated.

In step S704, D areas that correspond to the subject distance at one endin the subject distance range that is to be in-focus in the output imageare set as the first subject area by the control unit 101. Specifically,the control unit 101 specifies information indicating the representativedistance of the D area that corresponds to the position in the selectionreconstructed image where the face area that is the selected in-focussubject is located, and sets the D areas that have the samerepresentative distance information as the first subject area.

In step S705, the control unit 101 determines whether or not a secondin-focus subject selection was made. If the control unit 101 determinesthat a second in-focus subject selection was made, the procedure movesto step S706, and if it determines that a second in-focus subjectselection has not been made, the processing of step S705 is repeated.

In step S706, D areas that correspond to the subject distance at theother end in the subject distance range that is to be in-focus in theoutput image are set as the second subject area by the control unit 101.Specifically, the control unit 101 specifies information indicating therepresentative distance of the D area that corresponds to the positionin the selection reconstructed image where the face area that is thein-focus subject that was selected the second time is located, and setsthe D areas that has the same representative distance information as thesecond subject area.

In step S707, the control unit 101 determines whether or not anintermediate subject is included in the subject distance range that wasset; if an intermediate subject is included, the control unit 101 sets Dareas that correspond to the subject distance of the intermediatesubject as the intermediate subject area, and then subject selectionprocessing is completed. Specifically, for each of the face areasincluded in the subject distance range, the control unit 101 specifiesinformation indicating the representative distance of the D area thatcorresponds to that position in the selection reconstructed image, andsets the D areas that have the same representative distance informationas the intermediate subject area.

For example, consider the case where the selection reconstructed imageis the image shown in FIG. 8A. A selection reconstructed image 800includes a subject 801, a subject 802, and subjects 803 and 804 in orderof increasing distance from the digital camera 100. The distribution ofrepresentative distances here can be shown as a histogram for subjectdistances for focusing on subjects, as shown in FIG. 9.

Assume that the subject 801 was selected as the first in-focus subject,and the subject 803 was selected as the second in-focus subject. Here,the subject distance range that is to be in-focus in the output image isfrom 901 in FIG. 9 where the number of D areas that include the facearea of the subject 801 is counted, to 902 where the number of D areasthat include the face area of the subject 803 (and subject 804) iscounted. In the present embodiment, from among the D areas included inthis subject distance range, for each D area that includes a person'sface area and each D area that has the same representative distance asthat D area, an image generated so as to be focused at the correspondingsubject distance is used in output image generation. In the exampleshown in FIG. 8A, the subject 802 is an intermediate subject that isbetween the subject 801 and the subject 803. For this reason, for each Darea that has the same representative distance as the D areas thatinclude the subjects 801, 802, 803, and 804, the pixel group that wasgenerated so as to be focused at the subject distance that correspondsto the representative distance is used as pixels in the reconstructedimage that is to be output. Specifically, in the example shown in FIG.8A, the groups of D areas that have different cross-hatching accordingto the representative distances of the various areas as shown in FIG. 8Bare areas that are to be used in the generation of images of subjects tobe focused on in the output image. In FIG. 8B, an area 811 correspondsto the group of areas 901 (first subject area) in FIG. 9 where thenumber of D areas that include the face area of the subject 801 iscounted. Also, an area 812 corresponds to the group of areas 903(intermediate subject area) in FIG. 9 where the number of D areas thatinclude the face area of the subject 802 is counted. Furthermore, anarea 813 corresponds to the group of areas 902 (second subject area) inFIG. 9 where the number of D areas that include the face areas of thesubjects 803 and 804 is counted.

In this way, by executing the subject selection processing, it ispossible for the user to select subjects that are to be focused on inthe output image with an easy operation. In other words, even with LFdata obtained when shooting a scene that includes a large number ofsubjects that are to be focused on, it is possible to easily selectdesired subjects.

Note that in the present embodiment, it has been described that twotypes of subjects are selected, and then the range enclosed by thesubject distances of the two subjects that are to be focused on is setas the subject distance range that is to be in-focus in the outputimage. However, the implementation of the present invention is notlimited to this, and the subject distance range may be set using anothermethod. For example, a configuration is possible in which subjectdistance ranges that are not to be in-focus are set by the selection ofthe subject 803 in FIG. 8A and infinity (the background or far end) andthe selection of the subject 801 and the foreground (near end), and thenthe range not included in those ranges is set as the subject distancerange that is to be in-focus. Also, a configuration is possible in whichthe subject 801 and the subject 803 are selected, thereafter the subject802 is selected as a subject that is to be excluded from being anintermediate subject, and the divided ranges are set as the subjectdistance ranges that are to be in-focus. In other words, it issufficient that a range defined by the subject distances for focusing onat least two selected subjects is set as the subject distance range thatis to be in-focus.

After the subject distance range that is to be in-focus in the outputimage is set by executing the subject selection processing in this way,the control unit 101 moves to the processing of step S604.

In step S604, under control of the control unit 101, for each of thefirst, second, and intermediate subject areas included in the subjectdistance range, the image processing unit 109 generates a reconstructedimage that is focused at the subject distance that corresponds to thearea. In the case where the subject distance range described above withreference to FIG. 8A has been set, reconstructed images that are focusedat the subject distances corresponding to the areas shown in FIGS. 10Ato 100 are generated.

Also, the image processing unit 109 generates a reconstructed image thatis focused at a specified subject distance for an area not included inthe subject distance range. This specified subject distance may be thesubject distance that was set in the imaging optical system 104 duringshooting, for example. Also, from the viewpoint of making the subjectsincluded in the subject distance range prominent in the output image, itis preferable that the specified subject distance that is selected isdifferent from the subject distances of the subjects that are to befocused on in the subject distance range, such as a subject distancethat is not included in the subject distance range that was set.

Note that in the case of a D area in the vicinity of a boundary betweenareas, in order to mitigate discontinuity after the combining in thenext step S605, a reconstructed image may be generated for the subjectdistance that corresponds to a representative distance in the vicinityof the intermediate value of the representative distances of two Dareas.

In step S605, under control of the control unit 101, the imageprocessing unit 109 combines the reconstructed images for the variousareas that were generated in step S604, and generates an output image.Specifically, the image processing unit 109 performs combining such thatpixels of the reconstructed images generated as shown in FIGS. 10A to100 are used as pixels of corresponding areas in the output image, andthus the output image (FIG. 10D) is generated.

In step S606, the control unit 101 transmits the generated output imageto the display unit 110 and causes it to be displayed, and thusreconstructed image generation processing is completed.

Note that it is described in the present embodiment that for eachsubject included in the subject distance range set by two in-focussubject selections, a reconstructed image that is focused on thatsubject is generated, and then the generated reconstructed images arecombined. However, the implementation of the present invention is notlimited to this. For example, if there is a large difference in thesubject distances for focusing on the selected in-focus subjects, and anoutput image is generated so as to not be in-focus in areas that do notinclude the selected subjects, it is possible for the output image to bean image in which the focus has been adjusted unnaturally. In such acase, a configuration is possible in which, if the difference betweenthe subject distances for focusing on the selected in-focus subjects isgreater than a predetermined length, the control unit 101 performscontrol so as to cause the image processing unit 109 to generate animage that has a deep depth of field so as to be widely focused onsubjects in the shooting range. In other words, a reconstructed imagegenerated under the condition that a subject not included in the setsubject distance range is also to be focused on, may be generated as theoutput image. As described above, an image having a deep depth of fieldmay be an image generated using, from among the pixels in the LF data,pixels that correspond to light beams that passed through a limitedrange of divided pupil regions such as the central portion of the exitpupil 301.

As described above, the image processing apparatus of the presentembodiment can generate an image that is favorably focused on multipledesired subjects. Specifically, with this image processing apparatus,multiple subjects are selected in an image signal, and subject distancesfor focusing on each of the multiple selected subjects are specified. Asubject distance range defined by the specified subject distances isthen set. Then, based on the image signal, the image processingapparatus generates a reconstructed image that is focused on each of thesubjects that are to be focused on in the set subject distance range.

Other Embodiments

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment, and by a method, the steps of whichare performed by a computer of a system or apparatus by, for example,reading out and executing a program recorded on a memory device toperform the functions of the above-described embodiment. For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (e.g., computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2013-022643, filed Feb. 7, 2013, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising one ormore processors configured to function as: a first obtaining unit whichis able to obtain data from which images which are focused at aplurality of focus ranges can be generated after shooting and recordingthe data; a second obtaining unit which is able to obtain distanceinformation corresponding to a plurality of areas in the data; a settingunit which is able to specify focus positions corresponding to aplurality of subjects included in the data and set a focus rangedetermined by the specified focus positions for focusing the pluralityof subjects in an image; and a generation unit which is able to generatethe image by combining pixel values in the data such that all subjectscorresponding to the plurality of areas that the distance information isobtained by the second obtaining unit within the focus range set by thesetting unit are focused in the image, and wherein the focus range setby the setting unit is defined such that a focus position correspondingto one subject among the plurality of subjects is determined as one endof the focus range and a focus position corresponding to another subjectamong the plurality of subjects is determined as another end of thefocus range.
 2. The image processing apparatus according to claim 1,wherein the one or more processors further function as a display unitwhich is able to display, to a display, images corresponding to the dataand images generated by the generation unit.
 3. The image processingapparatus according to claim 2, wherein the one or more processorsfurther function as an instruction unit which is able to set theplurality of subjects.
 4. The image processing apparatus according toclaim 3, wherein the display is a touch panel display, and theinstruction unit sets, as the plurality of subjects, subjectscorresponding to operation inputs received on the touch panel display.5. The image processing apparatus according to claim 4, wherein thegeneration unit generates an instruction image that is focused on asubject at a predetermined focal position, and the instruction unitreceives an operation input of each of the plurality of subjects in theinstruction image that was generated by the generation unit.
 6. Theimage processing apparatus according to claim 1, wherein the data wascaptured by an image capturing apparatus and comprises a plurality ofviewpoint images.
 7. The image processing apparatus according to claim1, wherein the data was captured by an image capturing apparatus at onetime.
 8. The image processing apparatus according to claim 1, whereinthe data is light field data.
 9. The image processing apparatusaccording to claim 1, wherein the second obtaining unit obtains thedistance information by generating, for each of two different dividedpupil regions, an image using pixels that correspond to light beams thatpassed through the same divided pupil region in the data, and analyzinga distance for each subject included in the data based on a differencebetween the two types of generated images, and the setting unitspecifies the focus positions for focusing on each of the plurality ofsubjects with reference to a result of the analysis performed by thesecond obtaining unit.
 10. The image processing apparatus according toclaim 1, wherein the generation unit generates the image by combiningpixel values in the data in accordance with a pixel arrangement in afocal plane corresponding to the focus range set by the setting unit.11. The image processing apparatus according to claim 10, wherein thegeneration unit generates, for a region of each subject not within thefocus range included in the image, each pixel by adding pixel values ofthe data chosen such that a subject of the region is not focused. 12.The image processing apparatus according to claim 1, wherein, in a casewhere the focus range set by the setting unit is wider than apredetermined range, the generation unit generates the image in whichsubjects positioned at far end or near end are further focused.
 13. Theimage processing apparatus according to claim 1, wherein the generationunit specifies regions in which the subject in the focus range iscaptured, and generates the image such that each of the subjects of theregions is focused.
 14. An image capturing apparatus comprising one ormore processors configured to function as: an imaging unit which is ableto output data from which images which are focused at a plurality offocus ranges can be generated after shooting and recording the data; anobtaining unit which is able to obtain distance informationcorresponding to a plurality of areas in the data; a setting unit whichis able to specify focus positions corresponding to a plurality ofsubjects included in the data and set a focus range determined by thespecified focus positions for focusing the plurality of subjects in animage; and a generation unit which is able to generate the image bycombining pixel values in the data such that all subjects correspondingto the plurality of areas that the distance information is obtained bythe obtaining unit within the focus range set by the setting unit arefocused in the image, and wherein the focus range set by the settingunit is defined such that a focus position corresponding to one subjectamong the plurality of subjects is determined as one end of the focusrange and a focus position corresponding to another subject among theplurality of subjects is determined as another end of the focus range.15. A control method of an image processing apparatus, the controlmethod comprising: obtaining data which was captured by an imagecapturing apparatus, where each pixel of the data is corresponding to alight beam passed through a divided pupil region of an imaging opticalsystem; obtaining distance information corresponding to a plurality ofareas in the data; specifying focus positions corresponding to aplurality of subjects included in the data and setting a focus rangedetermined by the specified focus positions for focusing the pluralityof subjects in an image; and generating the image by combining pixelvalues in the data such that all subjects corresponding to the pluralityof areas that the distance information is obtained within the set focusrange are focused in the image, and wherein the set focus range isdefined such that a focus position corresponding to one subject amongthe plurality of subjects is determined as one end of the set focusrange and a focus position corresponding to another subject among theplurality of subjects is determined as another end of the set focusrange.
 16. A control method of an image capturing apparatus, the controlmethod comprising: outputting data from which images which are focusedwith a plurality of focus ranges can be generated after shooting andrecording the data; obtaining distance information corresponding to aplurality of areas in the data; specifying focus positions correspondingto a plurality of subjects included in the data and setting a focusrange defined by the specified focus positions for focusing theplurality of subjects in an image; and generating the image by combiningpixel values in the data such that all subjects corresponding to theplurality of areas that the distance information is obtained within theset focus range are focused in the image, and wherein the set focusrange is defined such that a focus position corresponding to one subjectamong the plurality of subjects is determined as one end of the setfocus range and a focus position corresponding to another subject amongthe plurality of subjects is determined as another end of the set focusrange.
 17. A non-transitory computer-readable recording medium recordinga program for causing a computer to execute a control method of an imageprocessing apparatus, the control method comprising: obtaining datawhich was captured by an image capturing apparatus, where each pixel ofthe data is corresponding to a light beam passed through a divided pupilregion of an imaging optical system; obtaining distance informationcorresponding to a plurality of areas in the data; specifying focuspositions corresponding to a plurality of subjects included in the dataand setting a focus range determined by the specified focus positionsfor focusing the plurality of subjects in an image; and generating theimage by combining pixel values in the data such that all subjectscorresponding to the plurality of areas that the distance information isobtained within the set focus range are focused in the image, andwherein the set focus range is defined such that a focus positioncorresponding to one subject among the plurality of subjects isdetermined as one end of the set focus range and a focus positioncorresponding to another subject among the plurality of subjects isdetermined as another end of the set focus range.
 18. A non-transitorycomputer-readable recording medium recording a program for causing acomputer to execute a control method of an image processing apparatus,the control method comprising: outputting data from which images whichare focused with a plurality of focus ranges can be generated aftershooting and recording the data; obtaining distance informationcorresponding to a plurality of areas in the data; specifying focuspositions corresponding to a plurality of subjects included in the dataand setting a focus range defined by the specified focus positions forfocusing the plurality of subjects in an image; and generating the imageby combining pixel values in the data such that all subjectscorresponding to the plurality of areas that the distance information isobtained within the set focus range are focused in the image, andwherein the set focus range is defined such that a focus positioncorresponding to one subject among the plurality of subjects isdetermined as one end of the set focus range and a focus positioncorresponding to another subject among the plurality of subjects isdetermined as another end of the set focus range.
 19. An imageprocessing apparatus comprising one or more processors configured tofunction as: a first obtaining unit which is able to obtain data fromwhich images which are focused at a plurality of focus ranges can begenerated after shooting and recording the data; a second obtaining unitwhich is able to obtain distance information corresponding to aplurality of areas in the data; a setting unit which is able to specifyfocus positions corresponding to a plurality of subjects included in thedata and set a focus range determined by the specified focus positionsfor focusing the plurality of subjects in an image; an instruction unitwhich is able to set the plurality of subjects; and a generation unitwhich is able to generate the image by combining pixel values in thedata such that all subjects corresponding to the plurality of areas thatthe distance information is obtained by the second obtaining unit withinthe focus range set by the setting unit are focused in the image, andwherein the focus range set by the setting unit is set such thatsubjects set by the instruction unit and another subject of which afocus position is between focus positions corresponding to the subjectsset by the instruction unit are focused.
 20. A control method of animage processing apparatus, the control method comprising: obtainingdata from which images which are focused at a plurality of focus rangescan be generated after shooting and recording the data; obtainingdistance information corresponding to a plurality of areas in the data;specifying focus positions corresponding to a plurality of subjectsincluded in the data and setting a focus range determined by thespecified focus positions for focusing the plurality of subjects in animage; setting the plurality of subjects; and generating the image bycombining pixel values in the data such that all subjects correspondingto the plurality of areas that the distance information is obtainedwithin the set focus range are focused in the image, and wherein the setfocus range set by the setting unit is set such that set subjects andanother subject of which a focus position is between focus positionscorresponding to the set subjects are focused.
 21. A non-transitorycomputer-readable recording medium recording a program for causing acomputer to execute a control method of an image processing apparatus,the control method comprising: obtaining data from which images whichare focused at a plurality of focus ranges can be generated aftershooting and recording the data; obtaining distance informationcorresponding to a plurality of areas in the data; specifying focuspositions corresponding to a plurality of subjects included in the dataand setting a focus range determined by the specified focus positionsfor focusing the plurality of subjects in an image; setting theplurality of subjects; generating the image by combining pixel values inthe data such that all subjects corresponding to the plurality of areasthat the distance information is obtained within the set focus range arefocused in the image, and wherein the set focus range set by the settingunit is set such that set subjects and another subject of which a focusposition is between focus positions corresponding to the set subjectsare focused.
 22. The image processing apparatus according to claim 1,wherein the data is data which was captured by an image capturingapparatus, where each pixel of the data is corresponding to a light beampassed through a divided pupil region of an imaging optical system. 23.The image capturing apparatus according to claim 14, wherein the data isdata which was captured by the imaging unit, where each pixel of thedata is corresponding to a light beam passed through a divided pupilregion of an imaging optical system.
 24. The control method according toclaim 15, wherein the data is data which was captured by the imagecapturing apparatus, where each pixel of the data is corresponding to alight beam passed through a divided pupil region of an imaging opticalsystem.
 25. The control method according to claim 16, wherein the datais data which was captured by the image capturing apparatus, where eachpixel of the data is corresponding to a light beam passed through adivided pupil region of an imaging optical system.
 26. The imageprocessing apparatus according to claim 19, wherein the data is datawhich was captured by an image capturing apparatus, where each pixel ofthe data is corresponding to a light beam passed through a divided pupilregion of an imaging optical system.
 27. The control method according toclaim 20, wherein the data is data which was captured by an imagecapturing apparatus, where each pixel of the data is corresponding to alight beam passed through a divided pupil region of an imaging opticalsystem.